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ABSTRACT 

The validities of the Pennsylvania Educational 
Quality Assessment (SQA) Preparing for a Changing World Instruments 
at grades 5, 8, and 11 were investigated. The study was carried out 
in a sjiiburban school district where many students had experienced a 
great ^deal of change in their lives. At each grade level 
approximately 60 students who had experienced a great deal of change 
and 60 students who had experienced little change participated. Each ^ 
student responded to the EQA instrument' appropriate to his/her grade 
level. At each grade level teachers were asked to choose students 
high and low in "emotional fortitude." In;^ investigating the 
validities of the instruments, EQA scores' of students rated high were 
compared with those of students rated low.. The study provided some 
evidence for the validity of the EQA instrument at each grade level. 
Validity support was gathered for the tptal instrument, for the 
Inef fective Solutions subtest, and for the Effective Solutions 
subtest; however, no validity support was found for the Emotional 
Adjustment subtest. In general, stronger validity support was found 
for the instruments when responded to^ by students who had experienced 
a great deal of change than when resp.onded to by students who had 
experienced little change. (Author/BJG) 
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Summary 

This study was performed as a means of investigating the validities of the 
Pennsylvania Educational Quality Assessment (EQA) Preparing for a Changing World 
Instruments at grades 5, 8 and 11. The study was carried out in a suburban school 
district where many students had experienced a great deal of change in their lives. 
At each grade level approximately 60 students who had experienced a great deal of 
whangt and 60 students who had experienced little change participated. Each student 
rei>ponded to the EQA instrument appropriate to his/her grade level. At each grade 
level teachers were asked to choose students high and low in "emotional fortitude." 
In investigating che validities of the instruments, EQA scores of students rated 
high were compared with those of students rated low. 

The study provided some evidence for the validity of the EQA instrument at 

each grade level. Validity support was gathered for the total instrument, for the 

/ 

Ineffective Solutions subtest and for t)ie Effective Solutions subtest. However, no 
validity support was found for the Emotional Adjustment subtest. In general, stronger 
*ai;.dity support was found for the instruments when responded to by students who had 

ex(jerienced a great deal of change than when responded to by students who had experi- 

I 

enced little change. 



Introduction 

In 1965 Pennsylvania's State Board of Education adopted 10 goals of quality 
edui^ation for the schools. of the Commonwealth. These goals reflect a concern for 
both cojj,nitive and noncognitive aspects of quality. They call :for Pennsylvania's 
schools to help each child to: (1) acquire the greatest possible self esteem; (2) * 
develop an understanding and appreciation of persons belonging to other social, 
cultural and ethnic groups; (3) achieve the highest possible mastery of the basic 
skills; (A) hold a positive attitude toward the learning process; (5) develop the 
habits ansi attitudes associated with responsible citizenship; (6) acquire good health 
habits; (7) have an opportunity to be creative in one or more fields of endeavor; (8) 
develop attitudes and knowledge leading to proper vocational development; (9) acquire 
the highest possible appreciation of human accomplishments; and (10) develop the 
ability to prepare for a world of rapid change. 

In the years following this statement by the State Board of Education, 
the Pennsylvania Department of Education has developed or acquired instruments to 
measure each of these goals at grades 5, 8 and 11. These instruments are used in 
the statewide assessment program carried out by the Division of Educational Quality 
Assessment (EQA) . 

Many of the goals stated by the Board of Education are difficult to measurie. 
One such goal is number 10 » Preparing for a Changing World. The instruments developed 
to measure this goal were developed by the Department's Division of Educational Quality 
Assessment with major responsibility assumed by Nolan. F. Russell. 

The Preparing for a Changing World instruments were designed t"* measure 
students' abilities to adjust to frustrating situations, such as those brought on by 
a change in one's life. The frustrating situations wer^ obtained from students in 
the instruments' developmental stages by asking them, to describe events they had 
experienced which necessitated soiQe form of adjustment and which were remembered as 
being difficult to deal with. 

4 



The instruments used in grades 5, 8 and 11 are similar in format, although 

» 

the items included differ somewhat for each grade level. The grade 8 instrument 
contains 35 items; the instruments employed in the other two grade leyels contain 
40 items. Each instrument is made up of three subtests. Emotional Adjustment, 
Effective Solutions and Ineffective Solutions'. Table 1 shows the/ most recent reli- 
abilities obtained for the instruments. 

In responding to the items of each instrument students are asked"* to indicate 
how they would react if they were placed in each of the frustrating situations 
included in the in.<>trument . One such situation is, ''My best friend told me that 
his/her dad was getting a job out of state and that they were moving." Listed under 
each situation are five ways students may react (e.g., looking for a new friend). 
Tne ways in which students may react are the items. Studen;ts respond to a four- 
point scale which depicts amounts of time they might ^^end pursuing each reaction. 
At grade 5 the scale is "a lot of time,** "some time," •^7ery little time" and "no 
time," and at grades 8 and 11 the scale is "a great deal of time," "some tiLme," 
"very little time" and "no time." Each' item is scored either from one to four or 
from four to one, depending on whether the reaction describes a favorable way to 
adapt to frustration or a way which is not favorable. 

The three subtests making up each instrument depict three aspects of ability 
to adjust to frustrating situations. The Emotional Adjustment subtest examines the 
amount of time students would spend "being upset." The less time students would spend 
being upset, the higher they would score on the Emotional>Adjustment subtests The 
Effective Solutions subtest concerns the amount of time students would spend pursuing 
f>o lilt ions which reflect positive adjustment. The more time students would spend 
pursuing solutions such as 'forking harder in school," the higher they would score 
on the Effective SoLui.ions subtest* In contrast the Ineffective Solutions subtest 
concerns the amount of time students would spend avoiding the use of aggressive or 
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withdrawing reactions* The less time students would spend pursuing solutions such 
as "dropping out of school,'* the higher they would score on the Inef fective^^Solutions 
subtest* 

'Table 1 

Reliabilities of the Educational' Quality Assessment Preparing 
for a Changing World Instruments^ 





Grade 5 




Grade 8 




Grade 11 


Subscale/ 
Total Scale 


'no* of 
Items 


Internai- 
Consis- 
■ tency 


No. of 
Items 


Internal 
Consis- 
tency 


Test- 
Retest 


No. of 
Items 


Internal 
Consis- 
tency 


Test- 
Retest 


Effective 
Solu tions 


15 


.75 


13 


.73 


.63 


^ 15 


.69 


.11 


Ineffective 
Sblu tions 


17 


.85 


13 


.82 


■ .71 


15 


.68' 


\.19 


Emotional 
AHjustment 


r 

8 


.79 


9 


.74 


.68 


10' 


.65 


■ .75 


Total Scale 


40 


.84 


35 . 


.-'83 

t 

t 


.79 


' 40 


.70 


.8') 



/ 

^At each grade level ^internal consistency reliabilities w,ere computed using the 
responses of approy^imately 3,500 students, randomly drawn from the students partici- 
pating in an assessment of one-third of the school districts in the state* Test- 
retesL reliabilities were computed using responses from approximately 400 8th grade 
and 400 11th grade students from a suburban school district* The interval between 
testings was 4 1/2 weeks* ' 



Methodology 



Theoretical Framework 



The approach taken in the validation study of the EQA Goal X instruments 
was a Concurrent one, providing information on how well scores on the instruments 
related to measures of student behaviors* A second purpose for carrying out the 
study was to help define more fully for school district personnel the types of 
behaviors to be expected from students obtaining the highest and lowest scores on 
the instruments* 



ERLC 



6 



In the study a measure of "emotional fortitude" was employed. Validity 
support for the Goal X instruments and for their subtests would be found if EQA 
scores were found to relate to "emotional fortitude," 

A measure of the amount of change experienced by students was also taken 
in order to acquire additional information about the Goal X instruments. The ques- 
tion addressed was whether the instruments had similar validities for students who 
had experienced a great deal of change in their lives and for students who had 
experienced little change. It was possible that, since students who had experienced 
change^ would have faced situations similar to those included in the EQA instruments, 
their predictions of their own behaviors would be more accurate than would be those 
of students who had experienced little change. 

Design of the Study — ^ 

In the sparing of 1974 all students participating in the study were asked to 
respond to the EQA Goal X instrument appropriate to their grade level. Information 
on two criterion measures was also gathered at this time. The two criterion measures 
were teacher ratings of students on emotional fortitude and school records indicating 
whether or not students had experienced change in their lives. School records were 
examined by counselors at grades 8 and 11 and the information was provided by class- 
room teachers at grade 5 (copies of the criterion measures are included in the 
Appendix) . 

The Teacher Rating Exercise described emotional fortitude as a trait having 
the following components: 

I. The ability to recove.r from a serious emotional setback, such as the 
death of a parent. 

II. The ability to confront difficult obstacle? which must be overcome. 

III. The ability to discipline and direct one's own behavior in order to 
achieve a goal. - ^ ^ , 

A list of each teacher's students was included at the bottom^ of the form. TJie teachers 

were instructed to select only those students whom they considered to be outstandingly 



low or high on the trait. Students were rated only by teachers by whom they had been 
taught during mo^t of the school yearT For grades 8 and 11 many teachers had contact 
wiih only a small proportion of the students in the study. Therefore, at these grade 
levels students were assigned to either the low or the high group if they were rated 
m this way by at least one teacher and if no other teacher rated them as candidates 
for the opposite group,. In grade 5, of course, only one teacher rated each .pupil. 
Students who were not rated low or highly were assigned to the middle group. 

The Criteria for Selecting the Change and the Non-Change Groups instrument 
wa& a survey form designed to be completed by school counselors about students 
assigned to them. The^ instrument consisted of an instruction page and two data forms. 
One data form consisted of students selected by counselors for the change group and 
the other form consisted of those selected for the nonchange group. Listed above the 
columns of the data. form for the change groups were the criteria for change. These 
included criteria such as death in the immediate family and residential movements 
resulting in a change of school. The appropriate columns were checked for each stu- 
dent listed. The data f orm Jor the nonchange group consisted only of a coiumjfi for 
listing names and a column for comments. 
Sample 

The sample consisted of 103 5th graders, 113 8th graders and 94 11th graders 
selected from a suburban Pennsylvania school district where many students had experi- 
enced change in their lives. This "high change" district was selected through 
examinations of 1970 census data, looking at such factors as divorce and separation 
rates, per cent of population who changed addresses and per cent of population under 

V* 

18 living with one parent or neither parent. 

Results 

The principal analyses of the study focused upon students assigned to the 

low and high emotional fortitude groups by teachers. At each grade level tl>ese groups 

/ 

^ t.fere compared on the Goal X instrument and subtests. For all analyses it was assumed 




that those assigned to the high emotional fortitude group should score higher than 
those assigned to the low group. The results of the independent t~tests are ^ 
reported in Table 2. The means enteringlnto these *tests are included in the 
Appendix • - ' ^ 

Table 2 



t-Vtilues Computed Between. Che High and Low Emotional 
Fortitude Groups on the Goal X Tests 









Grade 5 






Grade 


8 




Grade 


11 










Change 






Change 






Change 








Non- 


& Non- 




Non- 


& Non- 




Non- 


& Non- 






Change 


Change 


Cliange 


Change 


Change 


Change 


Change 


Change 


Change 






Group 


Group 


Groups 


Group 


Group 


Groups 


Group 


Group 


Groups 


Total 




.86 


! 

2.52* 


i.06 


2.65** 


-.52 


1.09 


.92 


-.30 


..26 


Emot . 


Adj. 


- .40 


1.28 


.3K 


1.67, 


-.49 


.88 


-1.25 


-.30 


t1.18 


Effec 


. Sol. 


1.71* 




.80 


.95 


-.09 


.50 


2.42** 


-.75 


/ 1.49 


Inef f ec. 
















I 




Sol, 




.39 


2.73** 


1.07 


2.48* 


-.55 


.96 . 


.56 


V .52 


.75 


Total 


minus 




















Emot • 


Adj. 


1.30 


2.20* 


2.34* 


1.87* 


-.41 


.84 


1.66*' 


-.20 


1.35 






22 


16 


38 


26 


18 


44 


41 


24 


71 



^Significant at the ,05 level ■ • 

**Significant at the ,01 level 

Fvir cvich column the N listed represents the total number of students, including both 
low and high rated students, involved in the analyses for that column. 



Validity support was found for the total test and the Ineffective Solutions 
subtest at grades 5 and 8, and the Effective Solutions subtest at giades 5 "an^ 11. 
The ETiotional Adjustment subtest did not receive any vali^dity support. In fact, the 
findings suggested that the Emotional Adjustment subtest had impaired the validity of 
the total test. Whereas two significant t-values were reported for the total test, 
four were reported for the total test minus the Emotional Adjustment subtest. 
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^\ As was anticipated as a possibility, stronger validity support was found 
for the instruments when responded to by students who had experienced change than 
when responded to by students who had experienced little change. The validity 
suppdVt found was strongest at grade 5, less strong at grade 8 and weakest at grade 
11- 

Examination of Student Reactions to the EQA Instruments 

As a means of investigating students' general impressions of the EQA instru- 
ments, grade 8 and 11 students who had responded to the instruments were asked to 
write answers to the following two questions: (1) Did you feel that you knew yourself 
well enough to predict what you would do in the situations?, and (2) Did you have any 
difficulty placing yourself in any of thfe situations? If sq, please describg them* 

The results found for question 1 provided validity support both for the EQA 
instruments and fdr self-report measurement in general. The overwhelming majority of 
students (92 per cent in grade 8 and 89 per ient in grade ll)-felt confident that they 
were capable of predicting how they would actually react if confronted by the situa^ 
tions reflected in the EQA instruments • 

In answering question 2 only a few students at each grade level indicated 
that they had difficulty in placing themselves in all of the situations of the instru- 
ments. However, at both grade levels many students had difficulty with one or more 
situations (32 per cent of grade 8 students and 66 per cent of grade 11 students had 
difficult^ with at least one situation). In examining student explanations about the 
situations which caused difficulty^, one consistent result was found. At both grade 8 
and grade 11 about one-fourth of the students had difficulty placing themselves in a 
situation in which a student was punished for smoking on school grounds. Students 
experrencing difficulty with this situation felt that, since they themselves would 
never smoke, it was difficult to relate to the situation. Perhaps because of the 
campaign being waged in this country to inform the public of the health hazards of 
smoking, the situation is becoming less relevant to students* lives than it was when 



Discussion and Conclusions 

Overall, some validity support was gathered at each grade level for the EQA 
, instruments. It cannot be said, however, that the results of the study were over- 
whelmlngly positive. There^ first of all, appeared to be one subtest, 'the Emotional 
Adjustment subtest, which impaired the validities of the instruments. Also >^ for none 
of the three grade levels was validity support gathered for the total instrument when 
respop^^ed to by both change and nonchange groups. Rather, at grades 8 and 11 no 
validity support was obtained for the instruments when responded to by students who 
had experienced littl^ change in their lives. 

One reason the study employed c^hange and nonchange groups was to test for 
just such an occurrence. The instruments^ call for students to predict how they wfll^ 
react in a variety of situations. If they have never experienced situations simil.ar 
to these it may be difficult for them to be accurate in their predictions and the 
validities of the instruments may suffer because of this. , ,^ 

One source of error in the study was the difficult rating jtask required of 
teachers. The strongest validity support for the instruments was gathered at grade 5 
This result may in part be du,e to the fact that, since teachers of grade 5 students 
had more contact with their students than did teachers of grade 8 arid grade 11 stu- . 
dents, their ratings of the emotional fortitude of their students were the most 
accurate o£ those obtained ^in the study. Also, the finding of stronger validity 
bupport for the instruments when responded to by change group students than when 
responded to by nonchange group students may in part be due to the abilities of 
teachers to rate change group students more accurately. Students who have experi- 
enccd change should reflect, in their day-to-day behaviors, their past abilities to 
deal with it. Those who have responded poorly would be expected to be weakened b^ 
it; those who have responded well would be expected to be strengthened. 
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Thus, the study described should be viewed as one attempt to assess the 
validities of instruments designed to measure a rather nebulous area. Other valida- 
tion studies of Pennsylvania's Preparing for a Changing World instruments and of 
smiliar instruments should be undertaken both to gatheir'lnlo'nnation about their 
validities and to help develop increasingly sophisticatad methods for assessing 
the validities of such instruments. ' / 



\ 



\ 



i 



Appendix 



ERIC 



13 



Criteria for Selecting the "Change" 
ana the "Non-Change" Groups 



In order to -validate EQA's Goal X instrument, "preparation for a 
rapidly changing world," two groups will be defined on the basis of change 
experience. The "change" group will include students who have experienced 
change in their lives, and the "non-change" group will include students who 
have not. We seek 60,. students for each of the two groups. These two groups 
will be formed at three grade levels to result in a tot^.l of 360 students. 
The three grade levels will be^ grades 5, 8, and 11. ^ 

Th^ criteria for selecting the "change" groups will include students 
who have experienced: 

(1) death in their immediate families , 

(2) the divorce of thfeir^ parents , 

(3) the separation of their parents, ^ 

(4) one or more residenxial movements resulting in a school 
change,'* and/or 

(5) :*ny other experiences significant enough to change students' 
lives . 

The criteria for selecting the "non-change" groups will include students who 
have not experienced any of the above situations. No other criteria should 
be employed. t , . 

In order to determine how the EQA test scores relate with change 
experience and emotional fortitude, we will need to know the names of the 
students. ^ 

! 

Also, we must knbw which criteria are used in assigning each student 
into the "change" groups. * For example, an eleventh grade student named John 
Doe was selected because his parents were divorced, and because he experi- 
enced two residential movements. 

Please enter the requested information on the enclosed data forms. 
One set of data forms is for the "change** group and the other is for the 
"non-chan&e" group. In selecting students into the "change" group, please 
select those who have experienced /the most change as defined by the 5 criteria. 

/ 



* If only one residential movement is noted for a given student, it bhould 
not be included unless it occurred within the past two years. 
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Teacher Rating Exercise 



We would like you to consider your students in regard to emotional 

fortitude • This is the trait of inward strength and maturity that enables 

individuals to endure substantial external and internal pressures. Following 
are three components of emotional fortitude: 

!• The ability to recover from a serious emotional setback, sucji 
as the death of a parent. 



II. The ability to confront difficult obstacles which must be 



overcome. 



^ r III« The ability to discipline and direct one's 
own behavior in order to achieve a goal* 

Attached is a list of students. assigned to you. Consider only 
those students who are outstandingly high or low on one or more of the 
characteristics . Place a letter "H" beside those you consider to be out- 
standingly high; and place a letter "L" 'beside those you consider to be 
outstandingly low. 

In the example below, students who are outstanding in regard to 
one or more components of emotional fortitude have been identified. 

Example: 



Atkinson, Doris 
Baker, Thomas 
Carter, Theodore 
Fot, Mary 
Mulphy, Sara 
Parker, Samuel 



L 
H 



Rice, Linda L 
Thomas, Henry 
Vaughn, Barbara 
Watson, Oliver H 



The remaining students are neither outstandingly high nor outstandingly low 
in regard to emotional fortitude. 



If ^ny of these students are not assigned to you please cross 
them off the list. 
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